{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\";\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Link Prediction With Graph Neural Networks\n",
    "\n",
    "In this example, we will use the Cora citation network [available for download here](https://linqs-data.soe.ucsc.edu/public/lbc/cora.tgz). Nodes are papers and links represent citations among papers. The objective is to use a small sample of positive (i.e., existing) links and negative (i.e, non-existing) links to build a model that can predict whther two nodes have a citation relationship.\n",
    "\n",
    "## STEP 1: Load and Preprocess Dataset\n",
    "Let's begin by loading and preprocessing the dataset.  By default, *ktrain* will holdout *10%* (i.e., `val_pct=0.1`) of the links for validation (along with an equal number of negative links).  An additional `train_pct` of links will be taken as the training set.  Here, we set `train_pct=0.1`, which is also the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: \n",
      "Type: Graph\n",
      "Number of nodes: 2708\n",
      "Number of edges: 5278\n",
      "Average degree:   3.8981\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0408 21:56:28.759857 140224628672320 utils.py:129] Note: NumExpr detected 32 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 8.\n",
      "I0408 21:56:28.761173 140224628672320 utils.py:141] NumExpr defaulting to 8 threads.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "** Sampled 527 positive and 527 negative edges. **\n",
      "** Sampled 475 positive and 475 negative edges. **\n"
     ]
    }
   ],
   "source": [
    "import ktrain\n",
    "from ktrain import graph as gr\n",
    "\n",
    "# load data with supervision ratio of 10%\n",
    "(trn, val, preproc) = gr.graph_links_from_csv(\n",
    "                                             'data/cora/cora.content', # node attributes/labels\n",
    "                                             'data/cora/cora.cites',   # edge list\n",
    "                                              train_pct=0.1, sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original graph: 2708 nodes and 5278 edges\n"
     ]
    }
   ],
   "source": [
    "print('original graph: %s nodes and %s edges' % (preproc.G.number_of_nodes(), preproc.G.number_of_edges()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "validation graph: nodes: 2708, links:4751\n"
     ]
    }
   ],
   "source": [
    "print('validation graph: nodes: %s, links:%s' % (val.graph.number_of_nodes(), val.graph.number_of_edges()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training graph: nodes: 2708, links:4276\n"
     ]
    }
   ],
   "source": [
    "print('training graph: nodes: %s, links:%s' % (trn.graph.number_of_nodes(), trn.graph.number_of_edges()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 2: Build a Graph Neural Network for Link Prediction\n",
    "Next, we build a graph neural network model.  *ktrain* currently supports [GraphSAGE models](https://cs.stanford.edu/people/jure/pubs/graphsage-nips17.pdf) for link prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "graphsage: GraphSAGE:  http://arxiv.org/pdf/1607.01759.pdf\n"
     ]
    }
   ],
   "source": [
    "gr.print_link_predictors()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "link_classification: using 'ip' method to combine node embeddings into edge embeddings\n"
     ]
    }
   ],
   "source": [
    "model = gr.graph_link_predictor('graphsage', trn, preproc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will wrap the model and data in a `Learner` object to facilitate training. For instance, let's set the global weight decay to 0.01."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner = ktrain.get_learner(model, train_data=trn, val_data=val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/amaiya/projects/ghub/ktrain/ktrain/core.py:266: UserWarning: recompiling model to use AdamWeightDecay as opimizer with weight decay of 0.01\n",
      "  warnings.warn('recompiling model to use AdamWeightDecay as opimizer with weight decay of %s' % (wd) )\n"
     ]
    }
   ],
   "source": [
    "learner.set_weight_decay(wd=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 3: Estimate Learning Rate Using Learning-Rate-Finder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "simulating training for different learning rates... this may take a few moments...\n",
      "Train for 29 steps\n",
      "Epoch 1/10\n",
      "29/29 [==============================] - 4s 122ms/step - loss: 1.1456 - accuracy: 0.6089\n",
      "Epoch 2/10\n",
      "29/29 [==============================] - 4s 121ms/step - loss: 1.0372 - accuracy: 0.5828\n",
      "Epoch 3/10\n",
      "29/29 [==============================] - 4s 130ms/step - loss: 1.0973 - accuracy: 0.5763\n",
      "Epoch 4/10\n",
      "29/29 [==============================] - 4s 124ms/step - loss: 1.1659 - accuracy: 0.6100\n",
      "Epoch 5/10\n",
      "29/29 [==============================] - 4s 126ms/step - loss: 0.9463 - accuracy: 0.5686\n",
      "Epoch 6/10\n",
      "29/29 [==============================] - 4s 125ms/step - loss: 0.5792 - accuracy: 0.7059\n",
      "Epoch 7/10\n",
      "29/29 [==============================] - 4s 127ms/step - loss: 0.4030 - accuracy: 0.8170\n",
      "Epoch 8/10\n",
      "29/29 [==============================] - 4s 124ms/step - loss: 0.5284 - accuracy: 0.8344\n",
      "Epoch 9/10\n",
      "29/29 [==============================] - 4s 123ms/step - loss: 0.4732 - accuracy: 0.8214\n",
      "Epoch 10/10\n",
      "29/29 [==============================] - 4s 121ms/step - loss: 0.5277 - accuracy: 0.7832\n",
      "\n",
      "\n",
      "done.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU1fnA8e87k31fCZBAFvaArAHZRFBxF9zaSl1wpbZabbW29qd1q62tWtu61a2tWndxQ1xQ2VW2sO8QwhaSQBIgIfsy5/fHDDFAEpKQm5tk3s/z5HHm3jsz79zS+86555z3iDEGpZRS3sthdwBKKaXspYlAKaW8nCYCpZTycpoIlFLKy2kiUEopL6eJQCmlvJyP3QE0V0xMjElKSrI7DKWU6lBWrlyZb4yJrW9fh0sESUlJpKen2x2GUkp1KCKyu6F9emtIKaW8nCYCpZTycpoIlFLKy1mWCETkPyJyQEQ2NLD/ahFZJyLrReR7ERliVSxKKaUaZmWL4FXg/Eb27wTONMacBvwReMnCWJRSSjXAslFDxphFIpLUyP7v6zxdCiRYFYtSSqmGtZc+gpuALxraKSIzRCRdRNLz8vJa9AH7i8r5ckMupZXVLY1RKaU6JdsTgYhMwp0IftfQMcaYl4wxacaYtNjYeudDnNSKXQe59Y2V7DlY2sJIlVKqc7J1QpmIDAZeAS4wxhRY+VlRwX4AHCyptPJjlFKqw7GtRSAiPYEPgWuNMdus/rzoYH/g2ERgjEFXaFNKeTsrh4++DSwB+olIlojcJCK3isitnkMeAKKB50VkjYhYWjciMtgXgEOeRLAjr5izn1rI37/ZbuXHKqVUu2flqKFpJ9l/M3CzVZ9/vMgg962hgpJKXC7DNa8sI6ewnDeX7uaOs3rj47S9u0QppWzhNVc/X6eD8EBfDpZUUlBSSU5hOWN7RVNQUsn3OyztnlBKqXbNaxIBuDuMD5ZUkltYDsC0UT0J9fdh9rpsmyNTSin7eGciKHIngsToIAb3CGdHXonNkSmllH28KhFEBh1tEZQB0DUsgKhgfx1SqpTyal6VCKLrtAicDiE6xJ/oYD8KiivsDk0ppWzjVYkgKsSPQ6WV5BwuJy7UH6dDiAzyo6i8mqoal93hKaWULbwrEQT5UVVjyMgrJi48wL0txD2s9JDeHlJKeSnvSgSeMhObsovo5kkE0UdLT5Q2LxEUV1Tz2ve7cLl0ZrJSqmPzykRQ7TLEhQUcs+1gcfMSwZcbcnlw1kaW7zrYukEqpVQb86pE0LdraO3jrsclgoJm3hrKOeweebQu63ArRaeUUvbwqkQQHxHIZ3eM59zUOM4eEAe0vCpptmdS2tq9ha0bpGrU+qxCyipr7A5DqU7F1jLUdhjYPZyXrkurfR4Z5IdI8xPB0bkIa/Zqi8BKR8qr+Grjfi4fHs+Rimoue/47bhqfTGyoP9Ehflw2TBe2U+pUeV0iOJ7TIUR4ahA1R46nRbDvcBl5RyqIDfW3Ijyv9+LCTJ6dn0H3iED8fIRql+GDVVkcLq0iIsiPSwZ353BZFZuyixiRGEmw/8n/SX+Xkc/zCzL46xWDSYgMaoNvoVT75vWJACDSM9HsqOoa10mrkeYWldO/ayhbco+wes8hzh3Y1eowm624oppn5m7nFxN7Ex7kW7u9xmXIOlTKhn1FVFTXcPnw9vWr2uUyfLoum5zCct5L3wvAnI259PP08eR7Ovbziyt4YNZGPl2TzZGKaiKCfPn8jjPoHhGIMYblOw/Sq0sIMSHuJP3p2mzu/WAdZVU1uAz897td/OHiVHYXlBAV7EdogG/9ASnVyXlVH0FDooP9KChxzy5evvMggx6aQ7anM7g+ZZU1HC6t4ryBXfH3cbAk05rqpaWV1Tz2xWZmrsxq8mveXbGHi59ZzI9fXMLM9L28uCiTV7/fdcwx97y/ljOfWMBtb63irvfWsq+R79oWjDE89dVW3lq2h8pqF/d9vJ4731nDX77YwgFPa+vLDblszT1CgK8Dfx8H43pHEx7oy1vL9jCgWxjPTBtGUVkV/1u6mxqX4dp/L+cnLy3lwVkbaz9n3pYDOB3Czyf24pwBXXg/fS+FZVVc8sy33PfRBhvPgFL20hYB7tXLduQVA7Axu5DyKhdr9h6me0RgvcfnePoHEqODGJkUxRILylgXFFdw9SvL2JJ7hPiIQK4YHo+INPqaV7/byUOfbiIlNpgN+4rYnFMEwJvLdvPzib3w83FQWFbF7PU5nD+wK5P6x/K7D9azZs9h4hv4rm3hYEklT8/LAGBpZgFzNuZy+fB4xvWKYdnOAkYmRXHPzHXMXpdN37hQHpoykPiIQL7atJ8dB4r5/YX98fdx8tm6HN5evodRyVF8m5FPfEQg8zYfoLSymiA/HzZmF5KWFMU95/Vn+c6DfLP5APd/vIGi8mq+2JBD3pFUvcWnvJK2CICu4QFkHy7DGFNbmfToRbQ+R8tYdwsPZEyvaLbkHiG/FesVuVyGG19dwc78Ei4fHs++w2W1iaohVTUunp2fwbje0Xx55wR6RAVypLyaAd3COHCkgq825QLw5YYcKqtd3DqxF5cNS8DPx8GavYdaLfaWONrfEhvqz6y12VRUu7j69J5cMSKBx68cwkWDuxEV7Ed+cSV9uoQyvGckcWEBXDs6kYemDMTfxwnADeOSOFxaxT3vryPA18EfLx1IWVUN87fkUV5Vw468EgZ2DwNgZFIk/eJC+XRtNj4OoarG1N6GUsrbaCIAEiIDKamsoaismgNF7gv65pwjDR6fU5sIAhjbKxpw/5JtLRuyC1mbVcgDl6Ry1+S+ACzYmtfoaxZuzSO/uJLrxybj5+PgqpE9AXjwklS6hQfw0ap9AMxam01SdBBDEsLx83EwqHuY7SOfjt6G+9Olg/BzOugRFcjwnpG1+4P8fLhpfDIAfeNCGnyf01OiObNvLPnFFZzVvwtn9u1CTIg/n67NZkvuEWpcpjYRiAjXjEkEYFzvGCb0jeWFhTvY7/khoJQ30UQAtbdFsg6X1l4ItuQ23CLYc7AUh7hbEoPiwwn0dZK+q/V+Vc/fkocInD+wKwmRQfTuEsLCbY0ngg9WZRET4sfEfrEA3DQ+mRevHcHpyVFMGdqdhdvy2He4jBU7DzE5Na72NtPQHpGsyypstOieMYbyKvfY/YwDjbdMWuJoYh3WM5InfjSYRy897YTbYNeNSWTq0O4n7ZT/vwsHEOjr5MoRCTgdwtSh3Zm7ZT/fbnefv4Hdw2uPvWxYPCmxwVwxIoGHpwykstrFI59uauVvp1T7Z+Xi9f8RkQMiUm8vnIj0F5ElIlIhIr+xKo6mONoXsO9QWe2toaxDZUx8Yj5PztlaexE8aktuEUkxwQT4OvF1OhjaI4L03a1XamLe1gMMSYgg2jPaZXzvGNJ3HWrwYu1yGb7NyOecAXH4ekY7Bfg6OW9gV0SEy4bFU+0yPPjJBiprXIztHVP72rG9oqnwXACNqb9u0jsr9nL6n+fy7oo9nPPUQuZszG217wruRODrFKKD/Zg6NJ4z+8aecExogC//vGoYyTHBjb5Xv66hrH/oXM7q754w+JORPaiqMTw3fwdRwX4kRP7QFxLi78O8uycyZUh3kmOC+dmEFD5bn8O2/Q23BpXqjKxsEbwKnN/I/oPAHcCTFsbQJPGei8O+w2UcKKqovdjsKijl2fkZ/Oe7ncccvznnCAO6htU+T0uKZHPOEUoqqk85lpzCMtZlHWZSvy6120YmRVFWVcPG7PpbKRl5xRwpryYtKare/f27hjEyKZJvNh/AxyGMrHPc2QO6MGNCCv9bupuvN+2v9/XLMgsoLKuqHVnz6ne7Wvjt6pdTWEbX8AAcjsY7w5uq7tDfvnGhDO8ZAcDffjyk0Q73G8YlE+jr5IUFO1olDqU6CssSgTFmEe6LfUP7DxhjVgBVVsXQVNHBfgT4Oti2v5jiimquHJHAo5cOIv3+c+gZFXRMf0FxRTV7DpbSv07dorSkKGpcplXutb+8aCcOES4fHl+7bWSS+375ip31n86jt6VGJEbWux/g9rP6ADCkRwQhdSZdiQi/Pa8fsaH+vO8Zplpd42J9ViGbc4pwuUzt9692GRKjg1iSWdCqv5pzDpfTLdy6UUsvXpvGN3efeUxyrU9ksB/TRvXkk7XZ7D1Yalk8SrU3HaKPQERmiEi6iKTn5TV+r7yF70/3iEBW73FfULtHBHDN6ERiQvxJjglmZ/4P98W3evoOBnT7oUUwrGcEIrBy96n1E+wvKuet5bu5dGg8PaJ+mPHaJSyAxOggFmfksyu/hMLSqmNu46zcfYjoYD+SohueJTuhTwxXjkjgmtE9T9jn43Rw+bB45m85wFNfb2PMX+ZxybPfcsE/F3P9qyvYkVfMxH6xjE6J4r/Xj8TPx8HrS3ad0netK7uwrLYsuBViQ/2bPDz2lgnJOAReXpxpWTxKtTcdIhEYY14yxqQZY9JiY0+8f9wa4iMC2ZLr/pV7tEQ14E4EeSW1F96jv477d/uhRRAW4EtSdHCjQ05PpqK6hp+/sRKHCL88q/cJ+09PjmLRtjwmPrmAIY98xbA/fs3GbHfBu5W7DzI8MbLR2x4iwpM/GtJgbZ4fpfXAAE/P3U7v2BD+edVQpo9JZNG2PKpdhiuGJ/DOjDGkxIYwZUh3Ply1j6LyU2/MuVyG/UXWtgiao1t4IJcPS+DdFXs53Mw1KpTqqHRCmUfdmjN1E0FKbDAllTXkHamgS1gAO/KKCfJznvALs3/X0GYnApfLsGDbASqrXRRX1LBqz2Ge/ekwkurpEL3nvP6M7RVDjctwqLSSp+du57n5GTwydRC7Ckq5atSJv/Sbo3eXEBb9dhJBvk4iPRVZx/aK4Y1le6hxGVK7/9ACum5MIjNXZvHhyiyuH5fc4s/ML67grvfWUlVj6B5hXYuguaaPTeLd9L18uGofN45v+fdTqqPQROBx47gkvtiQQ0WV65jbFEc7jjPzS+gSFsDBkkpiQ/1P+PU9oFsYX27MpaSiukmFzwAen7OVFxbuwOkQzhnQhcggXy46rVu9x8aG+nPpsB/6DQpKKnlx4Y7a8fZpjfQPNNXxyS021J9J/WJZsqOApOgfktPghAiG9ojg9SW7Gd8nlmB/5zG/6MuragjwdZ708z5Zk82ibXmcM6ALE/s2fv++LaV2D2NIjwjeXr6HG8YlnXRGt1IdnZXDR98GlgD9RCRLRG4SkVtF5FbP/q4ikgXcBdzvOSassfe0Up+4UNLvO4el/3c2QX4/XMiPJoKd+SWAuxxCZJDfCa/v3zUUY2BrMzpRl+8sIDTAhxqXYc7G/aQlRTX5onP92CScDuHJr7bi6xQGxYef/EUt8KfLTuP1m0bhPG5Ez/SxiWTml3Du3xdy82vptbfO5m7ez5CHv2KX53w15utNufSNC+GV6SPp2Uj/hh2mjezB9gPFrNpj76xrpdqClaOGphljuhljfI0xCcaYfxtjXjDGvODZn+vZHmaMifA8bvlN9lbg43QQHnhsBcru4YH4+TjI9JR4OFRaSWTQiVUqj3YeN+f20L7DZUweEEdYgDvxjGpg+Gd94sICmDo0nvIqF4Piw5v0C7wl4sICGJF4YlwXntaN+IhAIoL82JhdxGrPiKk5G3OpqHbxzgp3uYaMA8V8l5FPeVUNLpdh7ub9PDc/g0/XZrNil3tyW3t0yZDuBPs5eWuZlp1QnZ/eGjoJh0PoFRvCds+M2kMlVfSNCz3huITIQEIDfNjUwFj/41VWuzhwpIIeUUGc0TeWz9blMDK56YkA4JYzUpi5MuuYeQFtxd/HyVe/noDLGEb/eS63v7mK01OiWe4Z4jpz5V627z/C3C0HABiTEk1YoA9zNh47V+Hc1PZXvhsg2N+HqcPi+XBVFqcnR3HpsHj8fDrE2Aqlmk0TQRP0iwupvcAdKq0kqp5bQyLCkIQIVu9p2lyC3MJyjHFPZhudEk1ltau2Dk6T4+oayus3jrLsttDJHO0L+fXkvsxcmcVHq931jM7u34W5Ww6weu9h7p7cF5eBv3+zDYC7J/flurFJZOYVU1JRw5AeEbbE3hQ3jkviq425/PaDdezIL+b3FwywOySlLKGJoAn6dg3l4zXZ5B2poLSypnZUzfFGJEbyzLztFFdUHzNpqz5H1wCIj3BXMB3jKV7XXBPqKcfQ1m4+I4UbxyVz5Qvfs2rPYf5wcSq/u6A/yTHB+DoduFyGDdmFRAT6cvtZvRERhvU89c5tq/XuEsry/zuH38xcy3+/3cVPR/UkMbrxEhdKdUTa1m2Cfp5bQct2uiuM1tdZDO5E4DKw5rhWQVllDX/+fDOFpT+Mu6+bCDoDh0P4x0+G8cepA0mMDqJvXGht3SOHQ3j5ujSe+FHjJR7aI4dD+N35/XE44Om5GXaHo5QlNBE0wdE+gaOlpqOC61/S8OgM4+ML0C3bWcBLizJ5e8We2m1HSy93tXBGbVvrGR3EtWM633DLuLAArhrZk0/W7LN9NTelrKCJoAkSIgMJ9nOyLNN9gW+oRRAa4Eu/uNATSk0cLW39seceOrgrncaE+Fs22ke1rlsmpADw8iItPaE6H00ETSAi9O8WVjtyKKqBPgKAoT0iWL+v8JhaQPs9i91syT1Su87BroKS2qqnqv2Ljwjk0mHxvLNiDwWtuBqdUu2BJoImqjtEM6KBFgHAaQnhHC6tIuvQD7cQcovKCfJz4nQIH6/O5r0Ve1m28yAT+sQ0+D6q/bn1zBQqql28+v0uu0NRqlVpImii0Sl1E0H9fQQAQxLcwyHXZv3QYby/sJzE6GAm9Inhg1VZPDBrA+N6R3Pn2X2sC1i1ut5dQjk3NY7Xvt/FkVYouKdUe6GJoInqLvri62z4tPWNC8XP6WB9VmHtttyicrqGuWsF5R2pwBj46xWDj1lARXUMv5jYm6Lyat5evufkByvVQeiVqIlONi/gKD8fBwO6hbJ672FqXIacwjL2F1XQNTyAyalxxIT4c8fZfY6pdqo6jiE9IhjfO4bnF+wgX/sKVCehiaAZPvj5GP49Pe2kx53ZN5YVuw5yx9urGf/X+eQXV9AlNIAgPx+W/d/Z3DbpxPUGVMfx0JRUSitqeFgXuledhCaCZhiRGMXZA05eJO2aMYn4Ohx8tj6HGpd79NDR+QLHV/FUHU/vLqHMmJDCp2uza6vSKtWRaSKwQJfQAK4YkXBMJdOuYZ1n4phyL87j4xDeWLrb7lCUOmWaCCzy0JRU5t19JpcO7Q64F3lRnUeXsADOH9SV99L36ggi1eFpIrCIv4+T6BB//nLFYJ796bBmVxZV7d+MCSkcKa/mNZ1XoDo4TQQWC/B1cvHg7p2u/o5yL9l5dv8uvLx45zEFBZXqaDQRKHUK7j63H8UV1fz58812h6JUi2kiUOoUpHYP45YzUng3fa+ub6w6LCsXr/+PiBwQkQ0N7BcReVpEMkRknYgMtyoWpaz0y7N6Ex7oy4sLd9gdilItYmWL4FXg/Eb2XwD08fzNAP5lYSxKWSbY34drRyfy1ab9ZOYV2x2OUs1mWSIwxiwCDjZyyFTgdeO2FIgQkW5WxaOUlaaPTfLMK9AaRKrjsbOPIB7YW+d5lmebUh1ObKg/5w7sygersiivqrE7HKWapUN0FovIDBFJF5H0vLw8u8NRql5Xj+pJYVkVs9fl2B2KUs1iZyLYB/So8zzBs+0ExpiXjDFpxpi02NjYNglOqeYa0yua/l1DeX5BRm2NKaU6AjsTwSzgOs/oodFAoTFGf0qpDktEuPPsPmTmlTB7Xbbd4SjVZE0rst8CIvI2MBGIEZEs4EHAF8AY8wLwOXAhkAGUAjdYFYtSbeW8gV3p3zWUf87dzsWDu2u1WdUhWJYIjDHTTrLfALdZ9flK2cHhcLcKfv7mKmavy2bqUB3/oNq/DtFZrFRHct7ArvSLC+W5+Rm4f+8o1b5pIlCqlTkcwi0TUti2v5hF2/PtDkepk9JEoJQFpgzpTpdQf15ZnGl3KEqdlCYCpSzg5+Ng+tgkFm/PZ0tukd3hKNUoTQRKWeTq03sS6OvklcU77Q5FqUZpIlDKIhFBfvwoLYFP1uzjQFG53eEo1SBNBEpZ6MZxyVS7DK8v0UXuVfuliUApCyXFBDN5QBxvLNtNWaUWo1PtkyYCpSx2y4QUDpdWMXNVlt2hKFUvTQRKWSwtMZIhPSL4z7c7cWkxOtUOaSJQymIiwowzUtiZX8LnG7Suomp/NBEo1QbOH9SVPl1CeHT2Zs54fB7LMgvsDkmpWpoIlGoDTofw68l9yS0qZ9+hMp6dn2F3SErV0kSgVBu58LRuLP7tJO6a3JfF2/PJOHDE7pCUAjQRKNWmekQFMW1UT/x8HPz3u112h6MUoIlAqTYXHeLPpUO78+GqfRSWVtkdjlKaCJSyw/VjkymrquF/S3fZHYpSmgiUskNq9zDOTY3jmXkZ7MwvsTsc5eU0EShlk0cvHYS/j4N73l9LjU40UzbSRKCUTbqEBfDQlIGk7z7Eq9/vsjsc5cUsTQQicr6IbBWRDBG5t579iSIyV0TWicgCEUmwMh6l2pvLhsUzqV8sf/tqK/u1VLWyiWWJQEScwHPABUAqME1EUo877EngdWPMYOAR4DGr4lGqPRIRHpoykOoaw+NfbrU7HOWlrGwRjAIyjDGZxphK4B1g6nHHpALzPI/n17NfqU4vMTqY68Yk8tHqLHIKy+wOR3khKxNBPLC3zvMsz7a61gKXex5fBoSKSPTxbyQiM0QkXUTS8/LyLAlWKTtNH5uEAd5evvekxyrV2uzuLP4NcKaIrAbOBPYBJ6zeYYx5yRiTZoxJi42NbesYlbJcj6ggzuwbyzvL91BSUW13OMrLWJkI9gE96jxP8GyrZYzJNsZcbowZBtzn2XbYwpiUardun9SbvOIK/vLFFrtDUV7GykSwAugjIski4gdcBcyqe4CIxIjI0Rh+D/zHwniUatfSkqK4YWwy/1u6mw37Cu0OR3kRyxKBMaYauB2YA2wG3jPGbBSRR0RkiuewicBWEdkGxAF/sioepTqCX03uQ2iAD8/O0zLVqu34WPnmxpjPgc+P2/ZAncczgZlWxqBURxIW4MsNY5N4el4Gs9dlc/Hg7naHpLyA3Z3FSqnj3DwhheE9I7j9rdV8sV6XtlTW00SgVDsTFuDL2zNGM7B7GA9/uklHESnLaSJQqh3y93HyyNSB5BaV67KWynJNSgQicqeIhInbv0VklYica3VwSnmzEYlRXDE8gVcWZ5KZV2x3OKoTa2qL4EZjTBFwLhAJXAv8xbKolFIA3HtBfwJ8nPztq212h6I6saYmAvH890Lgf8aYjXW2KaUsEhvqzzVjEvliQw67dAEbZZGmJoKVIvIV7kQwR0RCAZd1YSmljrphXBI+DgcvLNxhdyiqk2pqIrgJuBcYaYwpBXyBGyyLSilVq0toAD89vSfvpe9lS26R3eGoTqipiWAMsNUYc1hErgHuB3QOvFJt5Ffn9CE0wJc/fbbZ7lBUJ9TURPAvoFREhgB3AzuA1y2LSil1jIggP34xsReLt+ezZq/WZVStq6mJoNoYY3AvHPOsMeY5INS6sJRSx7t6dCLhgb48M3e73aGoTqapieCIiPwe97DRzzwVQ32tC0spdbwQfx9mTEhh7pYDfLs93+5wVCfS1ETwE6AC93yCXNxrCzxhWVRKqXrdND6ZxOggHpi1gcpqHbinWkeTEoHn4v8mEC4iFwPlxhjtI1CqjQX4OnloykAy80r497c77Q5HdRJNLTHxY2A58CPgx8AyEbnSysCUUvWb1K8Lk1PjeHrudrIP62L36tQ19dbQfbjnEEw3xlwHjAL+YF1YSqnGPHBxKi5jdDipahVNTQQOY8yBOs8LmvFapVQr6xEVxG2TevPZ+hyW7zxodziqg2vqxfxLEZkjIteLyPXAZxy38phSqm3NmJBCTIgfz8zT4aTq1DS1s/ge4CVgsOfvJWPM76wMTCnVuABfJzefkaKTzNQpa/LtHWPMB8aYuzx/HzXlNSJyvohsFZEMEbm3nv09RWS+iKwWkXUicmFzglfK213jmWSmi92rU9FoIhCRIyJSVM/fERFptPqViDiB54ALgFRgmoikHnfY/cB7xphhwFXA8y3/Kkp5nxB/H24cl8w3m/ezKVsL0qmWaTQRGGNCjTFh9fyFGmPCTvLeo4AMY0ymMaYSeAd3iYpjPgI4+j7hQHZLvoRS3uz6sUmE+Pvw3AJtFaiWsXLkTzywt87zLM+2uh4CrhGRLNydz7+s741EZIaIpItIel5enhWxKtVhhQf5ct2YRD5fn0PGAV3SUjWf3UNApwGvGmMS8Kx+5qljdAxjzEvGmDRjTFpsbGybB6lUe3fT+GT8fRy8vCjT7lBUB2RlItgH9KjzPMGzra6bgPcAjDFLgAAgxsKYlOqUokP8uWxYPJ+s3UdhaZXd4agOxspEsALoIyLJIuKHuzN41nHH7AHOBhCRAbgTgd77UaoFrhmdSHmVi5mrsuwORXUwliUCY0w1cDswB9iMe3TQRhF5RESmeA67G7hFRNYCbwPXe9Y9UEo108Du4QzvGcGbS3ej/zdSzeFj5ZsbYz7nuBnIxpgH6jzeBIyzMgalvMm1YxL59btr+X5HAeN6611W1TR2dxYrpVrRBYO6ERXsx2vf77I7FNWBaCJQqhMJ8HVy9ek9+XrzfrbvP2J3OKqD0ESgVCdzw7hkAn2dPDdfJ5ipptFEoFQnExXsx3VjkvhkbbYWo1NNoolAqU7otkm9iA3x54FPNlDj0hFEqnGaCJTqhEIDfLnvogGsyyrknRV77A5HtXOaCJTqpKYM6c7olCge/3IreUcq7A5HtWOaCJTqpESERy8dRFlVDb//cL1OMlMN0kSgVCfWu0sovz2vH99s3s/MlVp6QtVPE4FSndyN45I5PTmKhz/dxJ6CUrvDUe2QJgKlOjmHQ3jyR0NwCFz/6nIKirW/QB1LE4FSXqBHVBAvX5dG1qEyzvvHYpZlFtgdkmpHNBEo5SVOT4nm41+MIzTAh9veWq0tA1VLE4FSXiS1exjPXz2corIqHpy10e5wVDuhiUApLzOgWxi3TuzF7HU5rNx9yDSC7BEAABYfSURBVO5wVDugiUApL/SzCSnEhvrz6GebdH6B0kSglDcK9vfhN+f2ZfWew3y2PsfucJTNNBEo5aWuHNGD/l1DeXLOVlxamM6raSJQyks5HcLPJ/ZiV0EpS3Q4qVezNBGIyPkislVEMkTk3nr2/11E1nj+tomIFk9Xqg2dN7ArYQE+vLtir92hKBtZtni9iDiB54DJQBawQkRmeRasB8AY8+s6x/8SGGZVPEqpEwX4OrlsWDxvL99L1qFSEiKD7A5J2cDKFsEoIMMYk2mMqQTeAaY2cvw04G0L41FK1WPGmb0QgSfmbLU7FGUTKxNBPFC3vZnl2XYCEUkEkoF5FsajlKpHfEQgt5yRwidrslm9R+cVeKP20ll8FTDTGFNT304RmSEi6SKSnpeX18ahKdX53TqxFzEh/jz62WadV+CFrEwE+4AedZ4neLbV5yoauS1kjHnJGJNmjEmLjY1txRCVUgAhnnkFK3cfYv7WA3aHo9qYlYlgBdBHRJJFxA/3xX7W8QeJSH8gElhiYSxKqZO4YkQC3cIDeGXxTrtDUW3MskRgjKkGbgfmAJuB94wxG0XkERGZUufQq4B3jLZHlbKVr9PB9LFJfL+jgPVZhXaHo9qQdLTrb1pamklPT7c7DKU6pcKyKiY9uYDE6CA+uHUsDofYHZJqJSKy0hiTVt++9tJZrJRqB8IDfbn/ogGs3nOYm19PJzOv2O6QVBvQRKCUOsZlw+L5zbl9Sd91kBteXcGR8iq7Q1IW00SglDqGiHD7WX349/UjyTpUpgvYeAFNBEqpeo1MiuLWM1P4cNU+XeO4k9NEoJRq0O2T+hAfEcgfPtlAVY3L7nCURTQRKKUaFOjn5MFLUtm2v5jXvt9ldzjKIpoIlFKNmpwax6R+sfz9623kFpbbHY6ygCYCpVSjRISHpgykymV49LNNJ3+B6nA0ESilTioxOpifTUhh9rocNucU2R2OamWaCJRSTXLT+GSC/Jy8vCjT7lBUK9NEoJRqkoggP34ysgez1mbzvyW7tFx1J6KJQCnVZHee3YexvWP4wycbueHVFRSW6azjzkATgVKqySKC/HjthpE8MnUg327P57HPN9sdkmoFmgiUUs0iIlw3Jombxifzzoq9rNyty1t2dJoIlFItcsfZfegeHsD9H2+gWmcdd2iaCJRSLRLs78MDlwxkc04Rry3ZbXc46hRoIlBKtdh5A92zjp/6aqvOOu7ANBEopVpMRHh4yiCqXYa/frnF7nBUC2kiUEqdkp7RQUwfm8Qna/axp6DU7nBUC1iaCETkfBHZKiIZInJvA8f8WEQ2ichGEXnLyniUUta4aXwyPg4HLyzaAUBhaRXfbNqPMYZvNu2norrG5ghVYyxLBCLiBJ4DLgBSgWkiknrcMX2A3wPjjDEDgV9ZFY9SyjpxYQFcmZbAzPQs9h4s5cbXVnDz6+k8MnsTN7+ezn+/22V3iKoRVrYIRgEZxphMY0wl8A4w9bhjbgGeM8YcAjDGHLAwHqWUhW6d0Itql4uLnl7Myt2HCPR11iaAN5bupsalJSnaKysTQTywt87zLM+2uvoCfUXkOxFZKiLnWxiPUspCPaODuHJEAgb451VDuW5sIgDDekaQdaiMbzbvb/XPfC99Lwu35bX6+3obn3bw+X2AiUACsEhETjPGHK57kIjMAGYA9OzZs61jVEo10Z8vO41Hpg4iwNfJ2F4xHCmv5u7JfbnyhSU89vlmzuwbS4Cvs1U+a/7WA/x25jr8nA5evWEkY3vHtMr7eiMrWwT7gB51nid4ttWVBcwyxlQZY3YC23AnhmMYY14yxqQZY9JiY2MtC1gpdWp8nI7aC31sqD9/vuw0okP8eWTqQHYVlPLiwtYpYf3lhhzufHs1/eJCSYgK5KevLOO3M9dqRdQWsjIRrAD6iEiyiPgBVwGzjjvmY9ytAUQkBvetIi12rlQnc0afWC4e3I3nFmSwK7/klN7rQFE5t721mqSYYF6+Lo2Pfj6O6WMSeS89izkbc1spYu9iWSIwxlQDtwNzgM3Ae8aYjSLyiIhM8Rw2BygQkU3AfOAeY0yBVTEppezzh4tT8XM6eOyLU6tYuiSzgBqX4c+XnUbP6CDCg3z5w8Wp9O8ayr0fruePszdRWlndSlF7B0vnERhjPjfG9DXG9DLG/Mmz7QFjzCzPY2OMucsYk2qMOc0Y846V8Sil7BMXFsAN45L4atN+Mg4Ut/h9lmYWEBbgw4BuYbXbfJwOnp42jLTESP773U5+8uJStu0/0hphewWdWayUajPTxybh53Tw1NdbcbVwOOnSzIOMSo7G6ZBjtveNC+WV6SN5+bo0dheUcP4/FvHK4kztN2gCTQRKqTYTE+LPbZN68/n6XO56bw1VzSxfnVtYzs78EkanRDV4zNkD4lh4zyTOTe3Ko59t5l8Ld/DWsj0s2KrTlBpi9/BRpZSX+eVZvXE6hCfmbOVwWRXPTBtGaIBvk167ao97EZy0pIYTAUBksB/PXz2cO99dw+NfbgXAIfDElUO4YkQCFdU1+Pu0zjDWzkATgVKqTYkIt03qTVSwH/d/vIGpz33HfRcO4OwBcSd97eo9h/DzcZBap3+gIQ6H8NcrTqOkoprBCeGs2HWQez9cx0er97Elt4g5v5pAdIj/Ca8rKK6g2mX4ZM0+Zq/L4YVrRtA9IrBF37WjkI52/ywtLc2kp6fbHYZSqhUs2VHA7z5Yx56Dpfzh4lTO7t+FhMhAfJz137X+0QvfU+0yfPSLcc3+rEMllVz49GJyCstxCFx9eiJ/vHTQMccYY7jk2W/ZmnuEqhr3tTEpOoh7L+jPVxv3c+c5fUiMDm7+F20HRGSlMSatvn3aR6CUss2YXtHMvftMzk2N44+zNzHxyQVc+cIS9h0uO+HYqhoX6/cVMqxHZIs+KzLYj7duGc0bN53OtaMTeWv5HpZmHjtafeXuQ2zYV8Sg+HAuGtyNd2aM5kh5Nbe+sYoPV+/jvo82dMrOZ701pJSyla/TwT+uGsoLCzPx93HwrwU7uOjpxTw8ZSCXDO6OwzM6aGvuEcqrXAztGdHiz0qOCSY5JpjBPcJZnJHPL95cxazbx5EQGQTA60t2Exrgw5s3n06Qn/vy+MWdZ/Dxmn2UV7l46uttfLxmH5cNSzj1L96OaItAKWW7ID8f7prcl9sm9Wb2L8eTEBnIne+sYfp/l9eOLDr66z0tsWUtgrrCAnx5+bo0qqpdzHh9JWWVNWTmFTN7XTZXjexRmwQAuoQFMGNCL34xsRdpiZHc99EGMg50rjkKmgiUUu1KUkwwn9w2ngcvSWXx9nz+9Jl7JvLi7fn0ig1utY7bXrEhPD1tGJtyinhm3nae+nob/j5OfnZmr3qP93E6ePanwwn0dXLrG6soqeg8s5c1ESil2h2nQ7hhXDI3j0/m1e938cbS3SzbWcAZfVq36OSk/l24bFg8/1q4g9nrcrj5jGRi6hlJdFTX8ACenjaMzLxi7n5vLVU1LqqbOReiPdJEoJRqt+69oD/jekdz/8cbKK9ycUaf1i81/bvz+9M1LICbxyfz63P6nvT4cb1juO+iVL7cmMtpD83hnKcWcqikstXjaks6fFQp1a4VV1TzwoIdrNl7mJeuG3HM/fvWYoxBRE5+YB0vLtzB5pwiPt+Qy8ikSF67YVSDw17bg8aGj2oiUEqpU/Be+l5+O3MdN49P5v6LU0/+Ag9jDK8v2c1Z/bvQIyrIwgjddB6BUkpZ5MdpPbh+bBKvfLuTj1ZnNfl1ew+W8eCsjZzx+HwOFJVbGOHJaSJQSqlTdN9FAxidEsXvPljPU19tpbyq5qSv2ZH/QynuM59YwHPzM6hpYUXWU6WJQCmlTpGv08HzV4/g7P5deHpeBg9/uvGkr8nMc6/U9t7PxjCxXyxPzNnKBf9cxHPzMzhSXmV1yMfQRKCUUq0gKtiPf10zgp9P7MXby/fy6drsRo/PzCsmPNCXkUmRPH/1cJ768RDCA315Ys5Wxj42j9+8v5b9ReV8ti6nSS2MU6ElJpRSqhXdNbkvyzIL+P2H6xmcEN5gkbrMvBJSYoNrRytdPjyBy4cnsHbvYd5ctpuPVu/jw1VZuAxMG9WTm8Yn4e/jtKRjWVsESinVinw9y2Y6BH759moqq+ufcJaZX0xKTMgJ24f0iODxK4fw5s2jmdivCxcP7sbby/dwzlOL+O93uyyJWVsESinVyhIig3j8ysHc+sYqHv9yywnDSosrqtlfVEFKbMMlrUclRzEqOYqK6hq6hgWQEBnIBad1syReS1sEInK+iGwVkQwRubee/deLSJ6IrPH83WxlPEop1VbOH9SNa0cn8sq3O5m3ZT9Abetgc04R4K53dDL+Pk7uvziV68clExcWYEmslrUIRMQJPAdMBrKAFSIyyxiz6bhD3zXG3G5VHEopZZf7LhpA+u5D/Ob9dYzvHcPi7Xm8PWM0czcfwMchjOkVbXeIgLUtglFAhjEm0xhTCbwDTLXw85RSql0J8HXy9FVDKamoZtbabEora5j+n+XMXpfN6SlRhAc2ba1mq1mZCOKBvXWeZ3m2He8KEVknIjNFpEd9byQiM0QkXUTS8/LyrIhVKaUs0SculH9eNZRfndOHj28bR2llDVmHypjchDWa24rdo4Y+BZKMMYOBr4HX6jvIGPOSMSbNGJMWG9u6ZWiVUspq5w/qxq/O6cuAbmG8cl0aE/rGctHg7naHVcvKUUP7gLq/8BM822oZY+ouGPoK8LiF8SillO1OT4nm9JT20TdwlJUtghVAHxFJFhE/4CpgVt0DRKTuWKgpwGYL41FKKVUPy1oExphqEbkdmAM4gf8YYzaKyCNAujFmFnCHiEwBqoGDwPVWxaOUUqp+uh6BUkp5AV2PQCmlVIM0ESillJfTRKCUUl5OE4FSSnk5TQRKKeXlOtyoIRHJA3bbHUczxQD5dgfRgen5O3V6Dk9NZzh/icaYekszdLhE0BGJSHpDw7bUyen5O3V6Dk9NZz9/emtIKaW8nCYCpZTycpoI2sZLdgfQwen5O3V6Dk9Npz5/2keglFJeTlsESinl5TQRKKWUl9NEoJRSXk4TgY1ExCEifxKRZ0Rkut3xdFQiEuxZ0/piu2PpaETkUhF5WUTeFZFz7Y6no/D8m3vNc+6utjueU6WJoIVE5D8ickBENhy3/XwR2SoiGSJy70neZiruJTyrgCyrYm2vWukcAvwOeM+aKNuv1jh/xpiPjTG3ALcCP7Ey3vaumefzcmCm59xNafNgW5mOGmohEZkAFAOvG2MGebY5gW3AZNwX9hXANNwrtD123Fvc6Pk7ZIx5UURmGmOubKv424NWOodDgGggAMg3xsxum+jt1xrnzxhzwPO6vwFvGmNWtVH47U4zz+dU4AtjzBoRecsY81Obwm4VVi5e36kZYxaJSNJxm0cBGcaYTAAReQeYaox5DDjhtoWIZAGVnqc11kXbPrXSOZwIBAOpQJmIfG6McVkZd3vRSudPgL/gvqh5bRKA5p1P3EkhAVhDJ7izoomgdcUDe+s8zwJOb+T4D4FnROQMYJGVgXUgzTqHxpj7AETketwtAq9IAo1o7r/BXwLnAOEi0tsY84KVwXVADZ3Pp4FnReQi4FM7AmtNmghsZIwpBW6yO47OwBjzqt0xdETGmKdxX9RUMxhjSoAb7I6jtXT4Jk07sw/oUed5gmebajo9h6dGz1/r8orzqYmgda0A+ohIsoj4AVcBs2yOqaPRc3hq9Py1Lq84n5oIWkhE3gaWAP1EJEtEbjLGVAO3A3OAzcB7xpiNdsbZnuk5PDV6/lqXN59PHT6qlFJeTlsESinl5TQRKKWUl9NEoJRSXk4TgVJKeTlNBEop5eU0ESillJfTRKAsJyLFbfAZU5pYsro1P3OiiIxtweuGici/PY+vF5FnWz+65hORpONLMNdzTKyIfNlWMam2oYlAdRieksD1MsbMMsb8xYLPbKwe10Sg2YkA+D86aH0fY0wekCMi4+yORbUeTQSqTYnIPSKyQkTWicjDdbZ/LCIrRWSjiMyos71YRP4mImuBMSKyS0QeFpFVIrJeRPp7jqv9ZS0ir4rI0yLyvYhkisiVnu0OEXleRLaIyNci8vnRfcfFuEBE/iEi6cCdInKJiCwTkdUi8o2IxHnKFd8K/FpE1ojIGZ5fyx94vt+K+i6WIhIKDDbGrK1nX5KIzPOcm7ki0tOzvZeILPV830fra2GJe8Wsz0RkrYhsEJGfeLaP9JyHtSKyXERCPZ+z2HMOV9XXqhERp4g8Ued/q5/V2f0x0OFX5VJ1GGP0T/8s/QOKPf89F3gJENw/QmYDEzz7ojz/DQQ2ANGe5wb4cZ332gX80vP4F8ArnsfXA896Hr8KvO/5jFTc9eQBrgQ+92zvChwCrqwn3gXA83WeR/LDLPybgb95Hj8E/KbOcW8B4z2PewKb63nvScAHdZ7XjftTYLrn8Y3Ax57Hs4Fpnse3Hj2fx73vFcDLdZ6HA35AJjDSsy0Md8XhICDAs60PkO55nARs8DyeAdzveewPpAPJnufxwHq7/13pX+v9aRlq1ZbO9fyt9jwPwX0hWgTcISKXebb38GwvwL1gzwfHvc+Hnv+uxL1kYH0+Nu61CTaJSJxn23jgfc/2XBGZ30is79Z5nAC8KyLdcF9cdzbwmnOAVPdaLwCEiUiIMabuL/huQF4Drx9T5/v8D3i8zvZLPY/fAp6s57Xrgb+JyF+B2caYxSJyGpBjjFkBYIwpAnfrAXct/aG4z2/fet7vXGBwnRZTOO7/TXYCB4DuDXwH1QFpIlBtSYDHjDEvHrPRvcrYOcAYY0ypiCzAvfQkQLkx5vjV2yo8/62h4X/DFXUeSwPHNKakzuNngKeMMbM8sT7UwGscwGhjTHkj71vGD9+t1RhjtonIcOBC4FERmQt81MDhvwb2417m0wHUF6/gbnnNqWdfAO7voToJ7SNQbWkOcKOIhACISLyIdMH9a/OQJwn0B0Zb9PnfAVd4+gricHf2NkU4P9Sgn15n+xEgtM7zr3Cv+AWA5xf38TYDvRv4nO9xlzkG9z34xZ7HS3Hf+qHO/mOISHeg1BjzBvAEMBzYCnQTkZGeY0I9nd/huFsKLuBa3OsZH28O8HMR8fW8tq+nJQHuFkSjo4tUx6KJQLUZY8xXuG9tLBGR9cBM3BfSLwEfEdmMe/3cpRaF8AHupQY3AW8Aq4DCJrzuIeB9EVkJ5NfZ/ilw2dHOYuAOIM3TuboJ9/38YxhjtuBeFjL0+H24k8gNIrIO9wX6Ts/2XwF3ebb3biDm04DlIrIGeBB41BhTCfwE93Koa4Gvcf+afx6Y7tnWn2NbP0e9gvs8rfIMKX2RH1pfk4DP6nmN6qC0DLXyKkfv2YtINLAcGGeMyW3jGH4NHDHGvNLE44OAMmOMEZGrcHccT7U0yMbjWQRMNcYcsisG1bq0j0B5m9kiEoG70/ePbZ0EPP4F/KgZx4/A3bkrwGHcI4psISKxuPtLNAl0ItoiUEopL6d9BEop5eU0ESillJfTRKCUUl5OE4FSSnk5TQRKKeXlNBEopZSX+3/2J7M4HNNFwgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.lr_find(show_plot=True, max_epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STEP 4: Train Model With [1Cycle](https://arxiv.org/pdf/1803.09820.pdf) Learning Rate Schedule]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "begin training using onecycle policy with max lr of 0.01...\n",
      "Train for 30 steps, validate for 33 steps\n",
      "Epoch 1/5\n",
      "30/30 [==============================] - 10s 334ms/step - loss: 0.8107 - accuracy: 0.6168 - val_loss: 0.5360 - val_accuracy: 0.7543\n",
      "Epoch 2/5\n",
      "30/30 [==============================] - 8s 275ms/step - loss: 0.5096 - accuracy: 0.8011 - val_loss: 0.4129 - val_accuracy: 0.8406\n",
      "Epoch 3/5\n",
      "30/30 [==============================] - 8s 282ms/step - loss: 0.3617 - accuracy: 0.8789 - val_loss: 0.4621 - val_accuracy: 0.8254\n",
      "Epoch 4/5\n",
      "30/30 [==============================] - 8s 278ms/step - loss: 0.3156 - accuracy: 0.9032 - val_loss: 0.4169 - val_accuracy: 0.8273\n",
      "Epoch 5/5\n",
      "30/30 [==============================] - 8s 282ms/step - loss: 0.2214 - accuracy: 0.9442 - val_loss: 0.4421 - val_accuracy: 0.8302\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f86fadecd68>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.fit_onecycle(0.01, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Predictions\n",
    "\n",
    "We will create a `Predictor` object and make predictions.  The predict method accepts a `networkx` graph with node features stored as node attributes and a list of edges (tuples of node IDs into the graph).  The model will predict whether each edge should exist or not.  Since we are making predictions existing edges in the graph, we expect to return a 1 (i.e., a string label of 'positive') for each input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = ktrain.get_predictor(learner.model, preproc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['positive', 'positive', 'positive', 'positive', 'positive'],\n",
       "      dtype='<U8')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict(preproc.G, list(preproc.G.edges())[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.save('/tmp/mylinkpred')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "reloaded_predictor = ktrain.load_predictor('/tmp/mylinkpred')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['negative', 'positive']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reloaded_predictor.get_classes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0.07637989521026611, 0.9236201],\n",
       " [0.21356898546218872, 0.786431],\n",
       " [0.1649041771888733, 0.8350958],\n",
       " [0.1919359564781189, 0.80806404],\n",
       " [0.12360215187072754, 0.87639785]]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reloaded_predictor.predict(preproc.G, list(preproc.G.edges())[:5], return_proba=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
